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(57) Method and apparatus for storing document 
images, for creating a retrieval index by which 
the document images may be retrieved, and for 
displaying the retrieved document images. 
When a document image is obtained, the docu- 
ment image is subjected to rule-based block 
selection techniques whereby individual reg- 
ions within the document region are identified, 
and the types of regions are also identified, 
such as title-type regions, text-type regions, line 
art-type regions, halftone-type regions and col- 
or image-type regions. The identification is 
used to create structural information and both 
the document image and the structural infor- 
mation is stored. A word-based retrieval index is 
created based on title-type regions and/or 
text-type regions, the retrieval index being used 
in conjunction with a search query so as to be 
able to retrieve documents which match the 
search query. The retrieved documents are dis- 
played in either a full image mode or a rapid 
browsing mode. In the rapid browsing mode, 
the full image of the document is not displayed, 
but rather only an abstract structural view of the 
document image based on the stored structural 
information. The level of abstraction may be 
specified by the operator in connection with the 
identified structural regions of the document, 
whereby, for example, only a structural view is 
displayed, or only title-type regions are dis- 
played mixed with remaining structure. 
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The present invention relates to a method and apparatus for creating a collection of indexed document 
images whereby the document images may be retrieved through the index, and to a method and apparatus 
for rapidly browsing through document images by viewing abstract structural views of the document images 
rather than the document images themselves. 

5 Recently, as increasingly larger storage devices have become available, and it has become possible to 

store documents not simply as ASCII text but also as a full facsimile image of the document. More specifically, 
it is now commonplace to convert a document into a computer-readable bit map image of the document and 
to store the bit map image of the document Accordingly, whereas ASCII text storage permitted storage and 
display of only text portions of documents, it is now possible to store a document in computer readable form 

w and to display not only the text but also pictures, line art, graphs, tables and other non-text objects in the docu- 
ment Likewise, it is also possible to store and display documents such that text attributes, such as size, font, 
position, etc., are preserved. 

Despite these advances, however, it is still difficult to retrieve the document images into computer memory 
quickly, and then to browse quickly through computer-displayed document images, for example, in a situation 

15 where a computer operator retrieves many document images and searches through those document images 
to find a particular document. These difficulties can be attributed to at least two limitations. First, current lim- 
itations on bandwidth of the computer systems limit of the speed at which documents may be retrieved from 
storage and displayed. For example, at 300 dots-per-inch resolution, an ordinary 8Y2 by 11 inch black and white 
document requires approximately 8.4 million bits to store a full document image. Adding halftone (grey levels) 

20 or color to the image, or increasing the resolution at which the image is stored, can easily increase storage 
requirements to many tens of millions of bits. The time required to retrieve those bits from storage and to create 
and display the resulting image is significant, even with current high speed computing equipment The time 
is lengthened even further in situations where the document image is retrieved from storage in a first computer 
and electronically transmitted, for example, by modem, to a second computer for display on the second com- 

25 puter. 

Second, even when a full document image is displayed to an operator, there is ordinarily too much infor- 
mation for an average operator to comprehend quickly. Much of the information displayed to the operator is 
not relevant to the operator's query and much time is therefore wasted in displaying the non-relevant infor- 
mation. And the presence of such non-relevant information can slow the operator in his attempt to locate and 
30 understand document information that is relevant to the query. 

Moreover, simply retrieving appropriate documents for presentation to an operator from a large collection 
of documents can prove difficult because of the large amount of information that must be searched. Conven- 
tional document retrieval systems ordinarily rely on the creation of a text index by which text documents may 
be retrieved. With document images (as opposed to text documents), it has been proposed to subject the docu- 
35 ment images to optical character recognition processing ("OCR processing") and to index the resulting text 
Systems such as those proposed in U.S. Patent 5,109,439 to Froessl suggest that is only necessary to OCR- 
process specific areas of the document to simplify the indexing process, but it has nevertheless heretofore 
proved difficult to create an adequate index for retrieval of document images. 
It is an object of the present invention to address the foregoing difficulties. 
40 In one aspect the invention provides a system for rapidly browsing through document images by displaying 

not the full document image, but rather only an abstract structural view of the document image. More specif- 
ically, a structural view of a document image includes labelled objects such as "title", "text", "table", "line art", 
"half-tone" and similar objects that are ordinarily found in documents, and those objects are displayed for pre- 
sentation to the operator in an abstract structured layout that is representative of the full document image. 
45 The level of abstract ion-and how the document is rendered on the display may be defined by the operator. 
Thus, it is possible for the operator to specify a high level of abstraction where only labelled objects are dis- 
played in the appropriate layout; likewise, it is also possible for the operator to specify a lower level of abstrac- 
tion where the original document image for only specific types of objects, such as for titles, is displayed. It is 
also possible for the operator to specify that text type objects should be displayed as ASCII equipment char- 
so acters rather than the original image of the text By displaying only abstract structural views of document im- 
ages rather than the full document image, and by permitting the operating to specify the level of abstraction, 
it is possible to retrieve and display document images quickly and to present the information to the operator 
in a quickly comprehendible format 

According to this aspect the invention provides a method and apparatus for storing document images in 
55 which a document image is obtained, for example, by scanning an original document the document image is 
processed to determine the structure of the document, and the structure of the document is stored together 
with the document image. Preferably, the structure of the document is determined by conventional block se- 
lection techniques which utilize a rule-based knowledge system for identifying specific areas in a document 
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and for determining the content of the image within those areas such that the document image is decomposed 
into a general set of objects. Suitable block selection techniques are described in co-pending application Serial 
No. 07/073,012 filed April 24, 1992 entitled "Method and Apparatus for Character Recognition". 

According to this aspect, the invention further provides a method for displaying document images in which, 

5 given an image of a document and its corresponding structure, a representation of the structure of the docu- 
ment is displayed and selected structural areas of the display are selectively replaced with corresponding 
areas from the full document image. The abstraction level at which the structure is displayed may be desig- 
nated, whereby the structural representation may be displayed with mixed areas of structure and full document 
image (or ASCII text equivalent) in accordance with the designated abstraction level. If desired, the display 

w may be either a visual display or a verbal display in which the structure and content of the document is enun- 
ciated through text-to-speech techniques which convert the structural and contextual information into spoken 
words for aural communication. 

In a further aspect of the invention, a retrieval index by which images of documents may be retrieved is 
created. According to this aspect a document is scanned to form a document image, and the document image 

15 is processed, for example, by block selection techniques, to identify areas on the document of a first type such 
as title areas, and areas on the document of other types. The document image in the first type areas are con- 
verted into text, for example, by optical character recognition ("OCR") techniques. The converted text is in- 
dexed so as to form a retrieval index, and the document image is stored together with the indexed text such 
that the stored document index may be retrieved with reference to the indexed text Advantageously, the in- 

20 ventton creates a retrieval index only from first-type areas of the document, such as from title areas of the 
document or from the first paragraph of text areas of the document 

This brief summary has been provided so that the nature of the invention may be understood quickly. A 
more complete understanding of the invention can be obtained by reference to the following detailed descrip- 
tion of the preferred embodiment thereof in connection with the attached drawings. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a perspective view of the outward appearance of an apparatus according to the invention, and 
Figure 2 is a block diagram of the Figure 1 apparatus. 
30 Figure 3 is a full diagram showing the overall operation of the Figure 1 apparatus. 

Figure 4 is a full diagram showing inputting and indexing documents according to the invention. 
Figure 5 is a view showing a representative input document 

Figure 6 is a view for explaining how various blocks identified within original documents are stored. 
Figure 7 is a flow diagram for explaining document display according to the invention. 
35 Figures 8 to 1 0 are views showing representative displays of an-abstracted document, in which Figures 9 

and 10 are views showing expansion of selected areas of the abstracted document 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

40 Figures 1 and 2 show an apparatus according to the present invention. 

As shown in these figures, reference numeral 1 0 designates personal computing equipment such as an 
IBM PC or PC-compatible computer. Computing equipment includes a CPU 11 such as an 80386 processor 
which executes stored program instructions such as operator selected applications programs that are stored 
in RAM 1 2 or specialized functions such as start-up programs or BIOS which are stored in ROM 14. Computing 

45 equipment 10 further includes a local area network interface 15 which provides interface to a local area net- 
work 16 whereby the computing equipment 10 can access files such as document files on a remote file server 
or send files for remote printing or have remote machines access document images on equipment 1 0 or other- 
wise interact with a local area network in accordance with known techniques such as by file exchange or by 
sending or receiving electronic mail. 

so Computing equipment 1 0 further includes a monitor 1 7 for displaying graphic images and a keyboard/mou- 

se 19 for allowing operator designation of areas on monitor 17 and inputting information. 

Mass storage memory 20, such as a fixed disk or a floppy disk drive, is connected for access by CPU 11 . 
Mass storage 20 typically includes stored program instruction sequences such as an instruction sequence for 
scanning, indexing, retrieving and displaying documents according to the invention, as well as other stored 

55 program instruction sequences for executing application programs such as word processing application pro- 
grams, optical character recognition programs, block selection applications programs, spreadsheet application 
programs, and other information and data processing programs. Mass storage memory 20 further includes 
document index tables which contain index information by which documents may be retrieved as well as bit 
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map images of documents, document structures, and ASCII text for text areas of the documents. Other data 
may be stored on mass storage memory 20 as desired by the operator. 

A modem 21, a facsimile interface 22, and a voice telephone interface 24 are provided so that CPU can 
interface to an ordinary telephone line 25. Each of the modem 21 , facsimile interface 22, and voice telephone 

5 interface 24 are given access to the telephone line 25 via a telephone line switch 26 which is activated under 
control by CPU 11 so as to. connect telephone line 25 to one of the modem 21, the facsimile 22, or the voice 
telephone interface 24, as appropriate to the data being sent and received on the telephone line. Thus, CPU 
11 can send and receive binary data such as ASCII text files or document images files via modem 21 and it 
can be controlled by a remote computer via modem 21, it can send and receive facsimile messages via fac- 
to simile interface 22, and it can interact on an ordinary voice telephone line via voice telephone interface 24. In 
this regard, voice telephone interface 24 is provided with a DTMF decoder 24A so as to decode tones on the 
voice telephone line 25 which correspond to operator depressions of a telephone keypad. In accordance with 
stored program instruction sequences in mass storage memory 20, the decoded tones are interpreted by CPU 
11 into operator commands, and those operator commands are executed so as to take predesignated actions 

15 in accordance with operator depressions of the telephone keypad. 

A conventional text-to-speech converter 27 is connector to the CPU 11. The text-to-speech converter 27 
interprets text strings that are sent to it and converts those text strings to audio speech information. The text- 
to-speech converter 27 provides audio speech information either to a speaker 27 for enunciation to a local 
computer operator, or provides audio speech information to the voice telephone interface 24 for enunciation 

20 over ordinary voice telephone lines. 

MIDI ("Musical Instrument Digital Interface") synthesizer 30 is also connected to CPU 11 and interprets 
MIDI music commands from CPU 11 so as to convert those MIDI music commands to audio wave forms. The 
audio wave forms are, in turn, played out over speaker 28 or provided to voice telephone interface 24 for play 
out over ordinary voice telephone lines. 

25 Scanner 31 operates to scan original documents printed on a sheet of paper, and to convert the information 

of those original documents into a bit-by-bit computer readable representation of that document. Scanner 31 
may be a simple black and white scanner, but more preferably scanner 31 includes at least half-tone (grey 
scale) processing capabilities and/or color processing capabilities. 

Printer 32 is provided to form images of documents under the control of CPU 11. Printer 32 may be an 

30 ordinary black and white printer, but, more preferably, printer 32 includes half-tone and/or color capabilities. 

A CD ROM 34, such as an optical disk, is connected for access by CPU 11. CD ROM 34 operates to sup- 
plement the storage in mass storage memory 20 and contains additional information concerning document 
indexes and document images and document structure. It is also possible to provide a write-once-read-many 
("WORM") optical device or an ordinary read/write optical device so as to further supplement the storage ca- 

35 pabilities. In addition, the local area network 1 6, CPU 11 can access document indexes and document images 
and document structure stored at remote file server locations, and via modem 21, CPU 11 can access docu- 
ment indexes and document images stored at centralized data base locations over ordinary voice telephone 
lines. 

Figure 3 is a flow diagram showing overall operation of the Figures 1 and 2 apparatus. The process steps 

40 shown in Figure 3 are executed by CPU 11 in accordance with stored program instruction sequences stored 
in mass storage 20 and executed out of RAM 14. 

In step S301, CPU 11 displays a main selection menu on monitor 17 and permits operator selection of 
one of the menu items. In step S302, if the operator has selected to input documents, then flow advances to 
step S303 in which document image processing is executed, after which flow returns to step S301. If the op- 

45 erator does not select document input but instead selects to retrieve documents (step S304) then flow advanc- 
es to step S305 in which document retrieval processing is executed, after which flow returns to step S301. If 
the operator does not select document retrieval but instead selects to display documents (step S306), then 
slow advances to step 5307 in which document display processing is executed, after which flow returns to step 
S301. Each of these processes is discussed in additional detail below. 

50 Although the flow processing illustrated in Figure 3 appears to be executed serially and sequentially, it is 

to be understood that the processes may be executed at significantly different times, in different order, and 
indeed may be executed by different machines. For example, it is possible for a first machine to input docu- 
ments and to store those documents for retrieval and display by a different machine. Likewise, it is possible 
for one machine to retrieve documents and transmit them to another machine, for example, via modem 21 , for 

55 display on that other machine. 

Figure 4 is a detailed flow diagram illustrating document input processing in accordance with step S303 
of Figure 3. 

In step S401 CPU 11 causes scanner 31 to scan an original document fed into scanner 31 so as to convert 
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the original document into a computer readable bit map image of that document. The bit map image may be 

temporarily stored in RAM 14 for further processing. 

In step S402, CPU 11 derives structural information concerning the layout of objects within the document. 

More specifically, CPU 11 subjects the bit map document image to block selection techniques such as those 
5 described in co-pending U.S. Application, Serial Number 07/873,012 so as to decompose the document image 

into a general set of objects. Using rule-based decisions, such block selection techniques are able to identify 

related regions or areas within the document, and are able to determine the type of images within those regions. 

For example, using the block selection techniques described in the aforementioned U.S. Application 

07/873,012, CPU 11 processes the scanned-in bit map image stored in RAM- 14 to identify related regions in 
10 the image. Then, CPU 11 identifies the type of region, for example, whether a region is a title region, a text 

region, a paragraph region, a table region, a line art image region such as a graph, a half-tone image region, 

a color image region, etc. 

Figure 5 illustrates regions identified on a document image in accordance with processing according to 
the process steps in Step 402. Figure 5 is a representative document image consisting of page 40 from a ma- 
ts gazine article. As shown in Figure 5, page 40 includes title regions such as regions 41, 42 and 47, as well as 
text regions such as regions 43, 44 and 48. Likewise, page 40 includes line art image regions 45, table regions 
49 and half-tone object regions 46. Other types of regions are possible. In accordance with processing in Step 
S402, CPU 11 identifies the related regions in the document and determines the type of image in each of the 
regions. In Figure 5, each region has been indicated diagrammatically with a circumscribing rectangle. 
20 Reverting to Figure 4, flow then advances to Step 5403 in which CPU 11 identifies the document type. 

More specifically, many documents have relatively standard structures; for instance, a business letter virtually 
always includes a corporate letterhead region (which will be identified in Step S402 as a line art image area), 
a date and an addressee region (which will be identified in Step S402 as separate but adjacent text regions), 
salutation, body and closing regions (which will be identified as text regions in Step S402), and a signature 
25 (which will be identified as a line art i mage region). CPU 11 stores templates of these standard document types, 
and in Step S403 compares the structural information derived in Step S402 with those stored templates. If a 
match is found between the stored template and the structural information derived in Step S402, then CPU 
11 assigns the associated document type to the document image stored in RAM 14. 

In Step S404, CPU 11 correlates related regions within the document. More specifically, there are often 
30 regions within a document that should be considered together, such as the text caption for a figure or the foot- 
note for a text block. For example, referring to Figure 5, text region 48 is related to title 47, and table region 
49 is related to title 50. Using rule-based decision processes, CPU 11 correlates related regions. 

In Step S405, CPU 11 subjects designated document regions to OCR processing in preparation for creating 
a retrieval index for the document. More specifically, so as to be able to retrieve the document image based 
35 on a word-based retrieval system, CPU 11 creates a retrieval index from text regions within the document It 
is possible to subject all text regions identified in Step S402 to OCR processing and to form the index based 
on the resulting text, but more preferably CPU 11 offers the operator a selectable level of indexes. More spe- 
cifically, CPU 11 offers the operator the option of selecting the index level in accordance with the following 
levels: 

40 Level 1: Title Regions Only 

Level 2: First Line Of Text Regions 

Level 3: Text Captions Related To Graphics 

Level 4: Full Text 

If the operator selects Level 1 (Titles Only"), then CPU 11 subjects only title regions to OCR processing. 

45 The resulting text from the title regions is provided for a retrieval index to allow the document to be retrieved 
when words in the title match a specified search query. Likewise, if the operator specif ies Index Level 2 ("First 
Line of Text Regions"), then CPU 11 subjects both title regions as well as only the first line of each text regions 
to OCR processing, and creates a retrieval index based on the resulting text. If Index Level 3 is selected (Text 
Captions Related To Graphics"), then CPU 11 subjects the items in Levels 1 and 2 to OCR processing as well 

so as text regions related to graphics as specified in the correlations obtained in step S404. Finally, if the operator 
specif ies- Index Level 4 ("Full Text"), then both title regions and full text regions wherever located are subjected 
to OCR processing and the resulting text provided to form the retrieval index. It is possible to include other 
indexing levels. 

In step S406, the ASCII text characters obtained from OCR processing are indexed and the document re- 
55 trieval index is updated appropriately. In addition to indexing the ASCII text words, a flag is also indexed which 
indicates the type of object from which the word was obtained, that is, title, text, caption, etc. The flag may be 
used as part of the retrieval process so as to retrieve documents based on the occurrence of words in particular 
regions of documents. 
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In Step S407, CPU 11 stores the structural information that was derived in Step S402 and that was cor- 
related in Step S406. Figure 6 is a representative view of how the structural information is stored. As shown 
in Figure 6, for each document the structural information includes a document identifier 51 which is the same 
identifier as that assigned to the full document image and by which it is possible to retrieve the full document 

5 image. In area 52, the document type derived in Step S402 is stored. At 53, all of the structural information 
for the document, and its layout within the document, is stored. As shown at 53, for each region identified in 
Step S402, CPU 11 stores a region identifier, a region type, rectangular coordinates that specify the position 
of the region on the page (here the upper left corner ("ULC") coordinates and the lower right corner ("LRC") 
coordinates), and all related regions identified in Step S406. In Figure 6, region 1 corresponds to region 41 in 

w Figure 5 and, as shown in Figure 6, includes a "title" type, upper left hand coordinates of <0,0>, lower right 
hand coordinates of <5,40>, and no related regions. The remaining regions illustrated in Figure 6 follow in like 
fashion. 

Region attributes 54 may also be stored, for example, font selection and size as illustrated in Figure 6. 
Other attributes may also be stored, such as number of text lines, number of paragraphs, etc. 

15 In Step S408, the document image is stored, preferably with OCR-processed document text The document 

text is stored in the event that a quick display of ASCII text characters is desired rather than the slower display 
of images of text regions. Accordingly, at the end of Step S408, CPU 11 has completed document input proc- 
essing and has stored a text-searchable index for the document, a full image for the document, the structure 
of the document, and, if desired, OCR-processed text of text regions of the document 

20 Reverting to Figure 3, if an operator selects document retrieval (Step S304), then flow advances to Step 

S305 for document retrieval processing. Document retrieval processing used in the preferred embodiment of 
the invention is a word-based retrieval and search system based on an operator-specified search query. Such 
a query may be specified by a logical sequence that must be found in each document (a "Boolean" search), 
or may be specified by a natural language based query system. The query may be made directly by a local 

25 operator, or remotely by an operator with voice or DTMF phone access, or from commands issued by a remote 
computer via modem or LAN. 

The operator may also specify that documents are retrieved based on the occurrence of words in particular 
document regions, such as title regions or captions. Documents matching the query are retrieved and are as- 
signed scores based on the degree that the document matches the query. Higher scores may be assigned for 

30 occurrences of words in particular regions of documents, the score being scaled the same as the above in- 
dexing level selection. Thus, for example, a higher score may be assigned for occurrences of words in title 
regions than for occurrences of words in text regions. When plural documents are retrieved, they are displayed 
in order of their score. 

When an operator selects document display processing (Step S306), then flow advances to Step S307 
35 for CPU 11 to execute document display processing. Figure 7 is a detail flow diagram illustrating document 
display processing. 

In Step S701 , CPU 11 allows the operator to set display options by which retrieved documents will be dis- 
played. Specifically, the operator selects one of two modes by which retrieved documents will be displayed: 
full document image mode or browse mode. If the full document image mode is selected, then full document 

40 images of retrieved documents will be displayed. If the browse mode is selected, then only an abstract struc- 
tural view of each document image is displayed, thereby permitting an operator to browse rapidly through all 
retrieved documents. In the browse mode, three further parameters may be set by the operator. First, the op- 
erator may select whether to display ASCII text characters of OCR-processed text regions or document images 
of text regions. Second, the operator may select the abstraction level at which the structural view is displayed. 

45 Nine abstraction levels are possible in this embodiment of the invention, though other arrangements are pos- 
sible: 



Level 1 
Level 2 
Level 3 

so Level 4 
Level 5 
Level 6 
Level 7 
Level 8 

55 Level 9 



Structure Only 
Titles 

Retrieval Parameters 
First Line of Text 
Full Text 
Tables 
Line Art 
Half-tone 
Color 

In accordance with the selected abstraction level, CPU 11 will display structural views of the document 
mixed with regions of full document images. If abstraction Level 1 is selected, then only structure is displayed. 
But if abstraction Level 2 or above is selected, then structural information is displayed mixed with regions of 
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full document images. For example, at abstraction Level 2, structural information is displayed mixed with im- 
ages (or ASCII text, if selected) of title regions of the document. The needed image regions are obtained from 
the full document image in accordance with the ULC and LRC coordinates stored with the structure. If ab- 
straction Level 3 is selected, then structural information is displayed mixed with images of titles (or ASCII text, 

5 if selected) and the retrieval parameters by which the document was retrieved. At abstraction Level 4, struc- 
tural information is displayed mixed with images of title regions, retrieval parameters, and the first line of text 
images in each text block. As before, if ASCII text display is selected, then ASCII text is displayed rather than 
images of text. Higher abstraction levels cumulatively display more and more of the full document image in 
accordance with the selected abstraction level. 

10 In addition to the abstraction level parameter, in the browse mode the operator can also select whether 

to allow image enhancement manually or automatically. Specifically, in manual image enhancement, when 
the structure of a retrieved document is displayed, the operator may target a particular region of the document, 
for example with mouse 19, and request for CPU 11 to replace the displayed region with the full document im- 
age. If automatic enhancement is selected, then CPU 11 automatically replaces the structural information that 

15 it is displaying with full document images. Automatic enhancement operates during quiescent periods of CPU 
operation, that is, when the CPU has sufficient processing time so as to enhance the image. Enhancement 
progresses in the order of the abstraction levels given above. Thus, during quiescent processing periods, CPU 
begins to replace, first, title regions with full images of the title regions, second, retrieval parameter regions 
with full images of the retrieval parameters, third, the first lines of the text regions with full images of the first 

20 lines of text, etc. 

After display options have been set in Step S701 , flow advances to Step S702 in which CPU 1 1 determines 
whether the full image mode has been selected. If the full image mode has been selected, then flow branches 
to Step S703 in which full images of the full images are displayed. 

On the other hand, if the browse mode has been selected, then flow advances to Step S704 in which the 

25 structure of the document is displayed at the selected abstraction level. A representative display of structural 
information is shown in Figure 8. In Figure 8, the operator has selected the browse mode, and has selected 
ASCII text display at abstraction Level 2, that is Titles". Thus, as shown in Figure 8, an abstracted structural 
view of the document is presented to the operator, with ASCII text equivalents of title regions. The remaining 
regions are simply labeled with the region type, as specified in the region type area of Figure 6. Labelling may 

30 be by color coding of the region, or, as shown in Figure 8, by applying a text label. Each of the regions is cir- 
cumscribed by a rectangular shape whose coordinates are specified by the "ULC" and "LRC" coordinates of 
Figure 6. ASCII text equivalents of title regions are displayed at the corresponding ULC and LRC coordinates. 
Thus, the operator is presented with an abstract structural view of the document with a selectable abstraction 
level. This permits the operator to browse quickly through retrieved documents so as to find particular docu- 

35 ments of interest 

Row then advances to step S705 in which CPU 11 determines if the manual enhancement parameter has 
been selected. If the manual enhancement parameter has not been selected, then flow branches to step S706 
in which CPU 11 automatically replaces structurally displayed regions of the document with full images (or 
ASCII text, if selected) of those regions during quiescent operating times of the CPU. Thus, assuming that 
40 sufficient quiescent operating time is available, if the automatic enhancement mode has been selected, then 
the full document will eventually be presented to the operator as the CPU replaces the structurally represented 
regions of that document At any time during this process the operator may select for the next retrieved docu- 
ment to be displayed (Step S710) whereupon flow returns to Step S702 so as to display the next retrieved 
document 

45 if the manual enhancement mode has been selected, then flow advances to Step S707 in which CPU 11 
determines whether a document region has been targeted by the operator. If no document region has been 
targeted by the operator, then CPU 11 does nothing, or more preferably, begins to replace structural regions 
of the document with full document images as shown at Step S708. Again, this process may be interrupted at 
any time by operator selection of the next document (Step S710). 

so On the other hand, if in Step S707 the operator targets a document region for replacement, then flow ad- 

vances to Step S709 in which CPU 11 retrieves the corresponding document image form the document image 
file and replaces the structurally represented region of the displayed document with the full document image. 
Thus, as shown in Figure 9, the operator has targeted area 54 which is a text region. CPU 11 thereupon retrieves 
the document image corresponding to the rectangular area circumscribed by the ULC and LRC coordinates, 

55 or the ASCII text equivalent if that option has been selected, and replaces the area in Figure 8 with the corre- 
sponding document image as shown in Figure 9. Flow then returns to Step S707 to await further designations 
by the operator. 

In Figure 9, the ASCII text equivalent for region 54 does not fit into the allotted area. In this situation, CPU 
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11 only displays the amount of text that will fit, but, so as to allow the operator to view all the text, CPU 11 
permits scrolling within the area. 

As further shown in Figure 9, the operator has requested for the first line of text from region 44 to be dis- 
played, and the ASCII text equivalent of that text is displayed at 55. Of course, if image mode was selected, 
5 then the image of the first line of text would have been displayed. 

In Step S709, if an operator targets a document region which has a related region, such as region 42 or 
43, then not only does CPU 11 replace the targeted region, but CPU 11 also replaces the related region as 
well. Accordingly, if the operator targets region 42, which is a title region, CPU 11 replaces title region 42 with 
the full image (or text) of region 42, and also replaces text region 43 with the full image (or text) of text region 
w 43. By virtue of this feature, the operator is provided with quick access to information that is desired without 
requiring the operator to execute additional keystrokes, etc. 

Figure 10 shows a situation in which the operator has selected for certain regions to be displayed as im- 
ages rather than as ASCII text More particularly, the operator has requested for full images of regions 46 and 
48, and those regions have been provided as shown at 56 and 57, respectively. 

15 



Claims 



1. A method for creating a retrieval index by which images of documents may be retrieved, the method com- 
20 prising the steps of: 

processing the document image to identify areas on the document of a first type and areas on the 
document of other types; 

converting the document image in the first type area into text; 
indexing the text; and 

25 storing the document image such that the stored document image may be retrieved by reference 

to the indexed text. 

2. A method according to Claim 1 , further comprising the step of storing the identity of the regions identified 
on the document. 



30 



35 



40 



45 



50 



55 



3. A method according to Claim 2, further comprising of the step of retrieving the stored identities and dis- 
playing a representation of the document in accordance with the stored identities. 

4. A method according to Claim 3, further comprising the step of progressive replacement of identified re- 
gions in the representation of the document with full images of the document 

5. A method according to Claim 4, wherein the progressive replacement is automatic. 

6. A method according to Claim 4, wherein the progressive replacement is manual in accordance with op- 
erator designation. 

7. A method according to Claim 3, wherein documents are displayed by score. 

8. A method according to Claim 1 , further comprising the step of selectably designating the first type of re- 
gion for indexing. 

9. A method according to Claim 8, wherein the first type of region is selected from the group consisting of 
title type regions and text type regions. 

1 0. Apparatus for creating a retrieval index by which images of documents may be retrieved, said apparatus 
comprising: 

a scanner for scanning an original document and for outputting a bit map document image; 

a first memory region for storing a document image and a retrieval index; 

a second memory region for storing process steps; and 

a processor for executing the process steps stored in said second memory; 

wherein said second memory includes process steps to (a) receive the bit map image scanned by 
said scanner, (b) process the bit map image to identify areas in the document of a first type and areas 
in the document of other types, (c) convert of the document image in the first type region into text, (d) 
index the text and update the index stored in said first memory with the indexed text, and (e) store the 
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15 



20 



25 



document image in said first memory such that the stored document image may be retrieved by reference 
to the indexed text. 

11. Apparatus according to Claim 1 0, wherein said second memory includes process steps to store the iden- 
tity of the regions identified in the bit map document in the bit map image in said first memory. 

12. Apparatus according to Claim 11, further comprising a display, and wherein said second memory includes 
process steps to display a representations of retrieved documents on said display in accordance with the 
identity of the document regions. 

13. Apparatus according to Claim 12, wherein said second memory includes process steps to progressively 
replace regions in the displayed document representation with corresponding regions from the full docu- 
ment image. 

14. Apparatus according to Claim 13, wherein the progressive replacement is automatic. 

15. Apparatus according to Claim 14, further including a designation means, and wherein the replacement 
is manual in accordance with designation by said designation means. 

16. A method according to Claim 12, wherein documents are displayed by score. 

17. Apparatus according to Claim 10, wherein the first type is selectably designatable. 

18. Apparatus according to Claim 17, wherein the selectable designations over the first type are selected 
from the group consisting of title type regions and text type regions. 

19. A document storage method comprising the steps of: 

scanning a document to form a document image; 

processing the document image to determine the structure of the document; and 
storing the structure of the document in correspondence with the document image. 

30 20. A method according to Claim 19, further comprising the step of indexing selectable types of document 
structure. 

21. A method according to Claim 19, further comprising the step of identifying a document type based on 
the determination of the document structure, and storing of the identified document type. 

55 

22. A method for displaying documents comprising the steps of: 

providing an image of a document and a corresponding structure for the document; 
displaying a representation of the structure of the document; 

replacing selected regions of the displayed document representation with corresponding document 
40 regions. 

23. A method according to Claim 22, further comprising the step of designating an abstraction level for dis- 
playing the representation of the document, and wherein in said displaying step the representation is dis- 
played with mixed areas of structure and document regions in accordance with the designated abstraction 

45 level. 

24. A method according to Claim 22, wherein in said providing step a plurality of document images and cor- 
responding document structures are provided, and further comprising the step of retrieving one of the 
plurality of document images and corresponding document structures for display. 

25. A method according to Claim 24, wherein the documents are displayed by score. 

26. A method according to Claim 24, wherein in said displaying step, a document structure is displayed mixed 
with retrieval parameters. 

55 27. A method according to Claim 22, further comprising the step of labelling the structural regions in the dis- 
played representation of the document. 

28. A method according to Claim 22, wherein said displaying step is a verbal enunciation of the document 
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45 



50 



55 



structure and/or content through text-to-speech conversion. 

29. A method according to Claim 22, further comprising the step of identifying document type based on the 
document structure. 

30. A method according to Claim 22, further comprising the step of identifying related regions in the docu- 
ment, and wherein in said replacing step, when a selected region of the displayed document representa- 
tion is replaced, related regions are also replaced. 

31. A method according to Claim 22, wherein in said replacing step, text type document regions are replaced 
with images of document regions or text equivalents in accordance with a manual selection. 



32. Apparatus for displaying a document comprising: 

means for providing an image of a document and a corresponding structure for the document; 
means for displaying a representation of the structure of the document; 
15 means for replacing selected regions of the displayed document representation with corresponding 

document regions. 

33. An apparatus according to Claim 32, further comprising designating means for designating an abstraction 
level for displaying the representation of the document, wherein said representation is displayed with 

20 mixed areas of structure and document regions in accordance with the designated abstraction level. 

34. An apparatus according to Claim 32, wherein a plurality of document images and corresponding docu- 
ment structures are provided, and further comprising retrieving means for retrieving one of the plurality 
of document images and corresponding document structures for display. 

25 

35. An apparatus according to Claim 34, wherein the documents are displayed by score. 

36. An apparatus according to Claim 34, wherein said displaying means displays a document structure mixed 
with retrieval parameters. 

37. An apparatus according to Claim 32, further comprising labelling means for labelling the structural regions 
in the displayed representation of the document. 

38. An apparatus according to Claim 32, wherein said displaying means is a verbal enunciation of the docu- 
ment structure and/or content through text-to-speech conversion. 

39. An apparatus according to Claim 32, further comprising identifying means for identifying the document 
type based on the document structure. 

40. An apparatus according to Claim 32, further comprising identifying means for identifying related regions 
40 in the document, and wherein when said replacing means replaces a selected region of the displayed 

document representation, related regions are also replaced. 

41. An apparatus according to Claim 32, wherein text type document regions are replaced with images of 
document regions or text equivalents in accordance with a manual selection. 



42. A document display apparatus comprising: 

a display; 

a first memory region from which an image of a document and corresponding structure for the 
document may be retrieved; 

a second memory region for storing processing steps; 

a processor for executing the process steps stored in said second memory; 

wherein said second memory includes process steps to (a) display on said display a representation 
of the structure of a retrieved document, and (b) replace selected regions of the displayed document rep- 
resentation with the corresponding document regions. 

43. Apparatus according to Claim 42, further comprising input means for inputting operator designations, and 
wherein said second memory includes process steps to permit designation of an abstraction level at which 
document representations are displayed, wherein document representations are displayed with mixed 
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areas of structure and document in accordance with the designated abstraction level. 

44. Apparatus according to Claim 42, wherein said first memory includes plural document images and plural 
corresponding structures, and wherein said second memory includes process steps to retrieve one of 

5 the plurality of stored document images and corresponding document structures for display. 

45. An apparatus according to Claim 44, wherein the documents are displayed by score. 



10 



46. Apparatus according to Claim 44, wherein document structure is displayed mixed with retrieval parame- 
ters. 

47. Apparatus according to Claim 42, wherein said processor labels the displayed document representation 
in accordance with the structure stored in said first memory. 

48. Apparatus according to Claim 42, wherein said processor enunciates the document structure in accor- 
ds dance with text-to-speech conversion. 

49. Apparatus according to Claim 42, wherein said processor identifies document type based on the stored 
document structure. 

20 50. Apparatus according to Claim 42, wherein the stored document structure correlates related regions in 
the document, and wherein when said processor replaces a selected region of the displayed document 
representation with corresponding regions of document image, said processor also replaces related re- 
gions. 
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40 



45 



50 



51. An apparatus according to Claim 42, wherein text type document regions are replaced with images of 
document regions or text equivalents in accordance with a manual selection. 

52. A data storage medium or signal conveying the information content of a plurality of documents, and further 
conveying indexing information including information identifying structural elements for each document 
and the contents of at least some of those elements, such that the general structure of a document can 
be retrieved for display without retrieving the complete information content of the document. 



55 
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42 ! Canon Wants Mutually 
L Rewarding Coexjstence j 

Source: rxTrtunc, 7/29/91 



43. + 



44. 



Ryuzaburo Kaku, Chairman of the Board of 
Canon, Inc. in his recent interview bad the following to 
{say about the corporate world The world is divided into 
four type* of companies: 

(1) Purely capitalistic enterprises that exploit 
their workers for profit 

(2) Those where management and labor work 
[closely together to maximize profits, but don't pay enough 
{attention to the community 

1 (3) A company that berth tries to make money 

■but also seeks to fulfill its corporate responsibiliiics to 
(society, but in a small scale way to a particular country or 
'region. 

} (4) A highly evolved type of company that 

icontributcs positively to world prosperity. 

^ Hnon is a$puing to be the fourth type of com- 

ipany. This is a company that is socially responsible and 
^Ujrnactices good corporate citizenship at home and overseas 
isnd thai can be referred to as a true global corporation. 
jWc have a basic philosophy to achieve a mutually reward- 
jing coexistence' among employees, shareholders, cus- 
tomers and the communities in which we do business. 



45 



Sala 



500 1- 



200 



too 



"90 



Y3 



V* 



Canon Develops World's First 
Ferroelectric Liquid Crystal Display 



Wall Street Journal, 10/2/91, 
Canon Press Release, 10/1791 



i 

J Source! 

I 
1 
i 

I In a news conference yesterday, Hiroshi Tanaka, a 
jCuum senior managing director said that the company has suc- 
jceeded n developing the world's first ferroelectric liquid crystal 
i (FLC) display screen. The screen will be test marketed next 
[spring in Canon's EZPS Japanese language DTP system. 
<i — — 
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Canon's Corporate Culture to Blend 
Best of U.S. & Japan 



l Source: Fortune, 806791 

J Mr. Hideharu Takcmoto, President of Canon 

iTJ-SA. was recently interviewed by Fortune Magazine 
J and had the following to say about Canon In North 
i America. " 
[ Mr. Takcmoto wants to create a new Canon 

i corporate culture. "The .best of American and Japanese 
[cultures must be blended to produce a richer corporate 
i alchemy— a new ideal." 

J Canon wants to create more jobs for 

i Americans in the 1990s and to make them an integral 
(part of the Canon famDy. Nfr. Takcmoto is further 
J committed to cultivating local talent and moving local 
(executives up through the ranks of Canon's highest cor- 
Jporate echelons. 
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Ink- Jet Printer Market Share 

Source: Computer Reseller Newrfnfo Corp ' 
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Source: Fortune, 8/26/91 
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Source: Fortune, 8/26/91 



. Mr. Hideharu Takemoto, President of Canon 
U-S JL was recently interviewed by fortune Magazine 
and had the following to say about Canon in North 
America. , — , 

Mr. Takcmoto wants to create a new Canon 
corporate culture. "The best of American and Japanese 
cultures must be blended to produce a richer corporate 
alchemy-a new ideal.* 

Canon wants to create more jobs for 
Americans in the 1990s- and to make them an integral 
part of the Canon family- Mr. Takemoto is further, 
committed to cultivating local talent and moving local 
executives up through the ranks ofCanon's highest cor* 
porate echelons. 
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